System for valuating users and user generated content in a collaborative environment

ABSTRACT

A system for valuating users and user generated content in a collaborative environment is described. The system may include a memory, an interface, and a processor. The memory may store an item, responses to the item, and ratings for each response. The processor may receive responses from the users based on the item provided by the content provider. The processor may receive ratings from the users for each of the responses. The processor may determine a response quality score for each response if the number of users who viewed the response satisfies a view threshold. The response quality score may be based on the ratings received for each response and the number of users who viewed each response. The processor may transform each response and the response quality score of each response into a graphical representation and may provide the graphical representation to the content provider.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 12/036,001, filed on Feb. 22, 2008, which is incorporated byreference herein.

TECHNICAL FIELD

The present description relates generally to a system and method,generally referred to as a system, for valuating users and usergenerated content in a collaborative environment, and more particularly,but not exclusively, to valuating users and user responses in acollaborative review system in order to identify the most valuable usersand the most valuable user responses.

BACKGROUND

Collaborative environments may allow users to cooperatively build off aninitial topic by structuring and restructuring the topic. The initialtopic may continually evolve as additional users provide insight to thetopic. The final result may be a representation of the group knowledgeover a period of time. However, collaborative environments may assumethat the insight and knowledge of all the users is equal. Collaborativeenvironments systems may be unable to properly account for users ofvarying knowledge and expertise on a given topic.

SUMMARY

A system for valuating users and user generated content in acollaborative environment may include a memory, an interface, and aprocessor. The memory may be connected to the processor and theinterface and may store an item, responses to the item, and ratings foreach response. The interface may communicate with devices of users and acontent provider. The processor may receive responses from the usersbased on the item provided by the content provider. The processor mayreceive ratings from the users for each of the responses. The processormay determine a response quality score for each response if the numberof users who viewed the response satisfies a view threshold. Theresponse quality score may be based on the ratings received for eachresponse and the number of users who viewed each response. The processormay transform each response and the response quality score of eachresponse into a graphical representation and may provide the graphicalrepresentation to the content provider.

Other systems, methods, features and advantages will be, or will become,apparent to one with skill in the art upon examination of the followingfigures and detailed description. It is intended that all suchadditional systems, methods, features and advantages be included withinthis description, be within the scope of the embodiments, and beprotected by the following claims and be defined by the followingclaims. Further aspects and advantages are discussed below inconjunction with the description.

BRIEF DESCRIPTION OF THE DRAWINGS

The system and/or method may be better understood with reference to thefollowing drawings and description. Non-limiting and non-exhaustivedescriptions are described with reference to the following drawings. Thecomponents in the figures are not necessarily to scale, emphasis insteadbeing placed upon illustrating principles. In the figures, likereferenced numerals may refer to like parts throughout the differentfigures unless otherwise specified.

FIG. 1 is a block diagram of a general overview of a system forvaluating users and user generated content in a collaborativeenvironment.

FIG. 2 is a block diagram of a network environment implementing thesystem of FIG. 1 or other systems for valuating users and user generatedcontent in a collaborative environment.

FIG. 3 is a block diagram of the server-side components in the system ofFIG. 2 or other systems for valuating users and user generated contentin a collaborative environment.

FIG. 4 is a flowchart illustrating the operations of determiningresponse quality scores in the systems of FIG. 1, FIG. 2, or FIG. 3, orother systems for valuating users and user generated content in acollaborative environment.

FIG. 5 is a flowchart illustrating the operations of determining a userresponse quality score in the systems of FIG. 1, FIG. 2, or FIG. 3, orother systems for valuating users and user generated content in acollaborative environment.

FIG. 6 is a flowchart illustrating the operations of determining thevalue of a user in the systems of FIG. 1, FIG. 2, or FIG. 3, or othersystems for valuating users and user generated content in acollaborative environment.

FIG. 7 is a flowchart illustrating alternative operations fordetermining a user response quality score in the systems of FIG. 1, FIG.2, or FIG. 3, or other systems for valuating users and user generatedcontent in a collaborative environment.

FIG. 8 is a flowchart illustrating alternative operations fordetermining and maintaining a user response quality score in the systemsof FIG. 1, FIG. 2, or FIG. 3, or other systems for valuating users anduser generated content in a collaborative environment.

FIG. 9 is a flowchart illustrating the operations of weighting ratingsprovided by users with user response quality scores in the systems ofFIG. 1, FIG. 2, or FIG. 3, or other systems for valuating users and usergenerated content in a collaborative environment.

FIG. 10 is a screenshot of a reporting screen in the systems of FIG. 1,FIG. 2, or FIG. 3, or other systems for valuating users and usergenerated content in a collaborative environment.

FIG. 11 is an illustration of a general computer system that may be usedin the systems of FIG. 2 or FIG. 3, or other systems for valuating usersand user generated content in a collaborative environment.

DETAILED DESCRIPTION

A system and method, generally referred to as a system, may relate tovaluating users and user generated content in a collaborativeenvironment, and more particularly, but not exclusively, to valuatingusers and user responses in a collaborative environment in order toidentify the most valuable users and the most valuable user responses.The principles described herein may be embodied in many different forms.

The system may allow an organization to accurately identify the mostvaluable ideas submitted in a collaborative environment bydifferentiating valuable ideas from popular ideas. The system utilizesan algorithm which incorporates the total number of unique viewers of anidea and the net rating of the idea to differentiate the valuable ideasfrom popular ideas. For example, the first ideas submitted to the systemmay have a high popularity rating by virtue of being first, theso-called “first movers” advantage. The system allows an organization todifferentiate between the ideas that are popular, such as these ideaswhich were submitted first, from ideas that are valuable. The systemalso implements dynamic thresholds to ensure the ideas are thoroughlyevaluated before being identified as valuable.

The system may also allow an organization to identify the most valuableusers in a collaborative environment. The organization may provideincentives to the most valuable users so as to encourage them tocontinue contributing to the collaborative environment. The system mayutilize an algorithm which incorporates both the quality and quantity ofthe users' contributions to determine which users are the most valuablein the collaborative environment. The quality of the user'scontributions may be weighted more than the quantity so as to identifyusers who submit many ideas, but few valuable ideas. The system mayfurther incorporate dynamic thresholds to ensure that only users whohave been thoroughly involved in the collaborative environment areidentified as valuable.

The system may be used in a collaborative environment to increase theaccuracy of the collaborative results. For example, in a collaborativeenvironment users may be presented with an initial item, such as aquestion, for review. A user may provide a response to the initial itemand may rate the responses of other users. The ratings of the users maybe used to determine which response is the most accurate response to theinitial item. The system may increase the accuracy determining the mostaccurate response by weighting the ratings of each user. The weight maybe indicative of the user's proficiency in the collaborativeenvironment. The weight for each user may be based on the user'sactivity in the collaborative environment and the ratings the user'sresponses have received from the other users in the collaborativeenvironment. Thus, when determining the most accurate response theweight applied to the ratings of an expert user may be higher than theweight applied to the ratings of a novice user. By applying more weightto the ratings of the expert users and less weight to the rating of thenovice users, the system may increase the accuracy of the collaborativeresults.

FIG. 1 provides a general overview of a system for valuating users anduser generated content in a collaborative environment 100. Not all ofthe depicted components may be required, however, and someimplementations may include additional components. Variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the claims as set forth herein. Additional,different or fewer components may be provided.

The system 100 may include one or more content providers 110A-N, such asany providers of content for review, a service provider 130, such as aprovider of a collaborative environment, and one or more users 120A-N,such as any users in a collaborative environment. For example, in anorganization the content providers 110A-N may be upper management, ordecision makers within the organization while the users 120A-N may beemployees of the organization. In another example, the content providers110A-N may be administrators of an online collaborative web site, suchas WIKIPEDIA, and the users 120A-N may be any web surfers providingknowledge to the collaborative website. Alternatively or in addition theusers 120A-N may be the content providers 110A-N and vice-versa.

The initial item may be any content capable of being responded to by theusers 120A-N, such as a statement, a question, a news article, an image,an audio clip, a video clip, or generally any content. In the example ofan organization, a content provider A 110A may provide a question as theinitial item, such as a question whose answer is of importance to theupper management of the organization.

The users 120A-N may provide responses to the initial item, such ascomments, or generally any information that may assist the collaborativeprocess. The users 120A-N may also provide ratings of the responses ofthe other users 120A-N. The ratings may be indicative of whether theusers 120A-N believe the response is accurate for the initial item. Forexample, if the initial item is a question the users 120A-N may rate theresponses based on which response they believe is the most accurateresponse to the question.

In one example the ratings provided by the users 120A-N for a responsemay be either an indicator that the users 120A-N like, or agree with,the response, such as a positive rating or “thumbs up” indicator, or anindicator that the users 120A-N dislike, or disagree with, the response,such as a negative rating or “thumbs down” indicator. The serviceprovider 130 may use the ratings provided by the users 120A-N for aresponse, and the number of users 120A-N who have viewed the response,to generate a response quality score for the response. For example, theresponse quality score of a response may be determined by subtractingthe number of negative ratings from the number of positive ratingsreceived for a response and dividing the result by the number of uniqueusers 120A-N who viewed the response. Alternatively, the result may bedivided by the number of unique users 120A-N who rated the response.

The service provider 130 may order the responses based on the responsequality scores, and may provide the ordered responses to the contentprovider A 110A who provided the initial item. The service provider 130may only provide responses to the content provider A 110A if theresponses have been viewed by enough of the users 120A-N for theresponse quality scores to be deemed substantial. The service provider130 may identify a view threshold, and may only provide response qualityscores for responses which satisfy the view threshold. For example, theservice provider 130 may only provide response quality scores for theresponses which are in the upper two-thirds of the responses in terms oftotal views by the users 120A-N. In this example, if there are threeresponses, two which were viewed by ten users 120A-N, and one which wasonly viewed by eight users 120A-N, the service provider 130 may onlygenerate a response quality score for the responses which were viewed byten users 120A-N. By omitting response quality scores for responses witha small number of views, the service provider 130 can control forsampling error which may be associated with a relatively small sampleset. The steps of determining response quality scores are discussed inmore detail in FIG. 4 below.

The service provider 130 may provide the content provider A 110A with alist of responses which satisfy the view threshold, ordered based on theresponse quality scores. The list of responses may be provided to thecontent provider A 110A in a graphical representation, such as the listof responses illustrated in FIG. 10 below. The graphical representationmay assist the content provider A 110A in quickly reviewing theresponses with the highest response quality scores and selecting theresponse which the content provider A 110A believes is the mostaccurate. The content provider A 110A may provide an indication of theirselection of the most accurate response to the service provider 130.

The service provider 130 may maintain a user response quality score foreach of the users 120A-N in the collaborative environment. The userresponse quality score may be indicative of the level of proficiency ofthe users 120A-N in the collaborative environment. The user responsequality score of a user A 120A may be based on the response qualityscores of the responses provided by the user A 120A. For example, theuser response quality score of a user A 120A may be the average of theresponse quality scores of the responses provided by the user A 120A.The service provider 130 may only determine user response quality scoresof a user A 120A if the number of responses provided by the user A 120Ameets a contribution threshold. For example, the service provider 130may only determine the user response quality score for the users 120A-Nwho are in the upper two-thirds of the users 120A-N in terms of totalresponses contributed to the collaborative environment. In this example,if a user A 120A contributed ten responses, a user B 120B contributedten responses, and a user N 120N contributed eight responses, then theservice provider 130 may only determine a user response quality score ofthe user A 120A and the user B 120B. By excluding the users 120A-N withlow numbers of contributions, the service provider 130 can controlsampling error which may be associated with a relatively small number ofcontributions. The steps of determining user response quality scores ofthe users 120A-N in this manner are discussed in more detail in FIG. 5below.

Alternatively or in addition, the user response quality score for theuser A 120A may be based on the number of responses the user A 120A hascontributed to the collaborative environment, the number of times theresponses of the user A 120A have been viewed by the other users 120B-N,the average rating the users 120B-N have given the responses of the userA 120A, and the number of responses of the user A 120A has been selectedas the most accurate response by one of the content providers 110A-N.The user response quality score may be normalized across all of theusers 120A-N. For example, if the user response quality score is basedon the number of responses provided by the user A 120A, the serviceprovider 130 may divide the number of responses provided by the user A120A by the average number of responses provided by each of the users120A-N to determine the user response quality score of the user A 120A.The steps of determining the user response quality scores of the users120A-N in this manner are discussed in more detail in FIGS. 7 and 8below.

The service provider 130 may use the user response quality score as aweight in determining the total ratings of the responses by multiplyingthe user response quality score by each rating provided by the user A120A. If the service provider 130 requests the users 120A-N to ratewhether they “like” or “don't like” a response, a “like” rating maycorrelate to a value of 1 and a “don't like” rating may correlate to avalue of 0. The rating given by each of the users 120A-N may bemultiplied by the normalized user response quality score of each of theusers 120A-N to determine the weighted rating of each user. The weightedrating of each of the users 120A-N for a given response may then beadded together to generate a total rating for the response. Bymultiplying the ratings of the users 120A-N by a normalized weight, theratings of the more proficient users 120A-N may be granted a greateraffect than those of the less proficient users 120A-N. The steps ofweighting responses based on user response quality scores are discussedin more detail in FIG. 9 below.

The service provider 130 may also determine a user value for each of theusers 120A-N in the collaborative environment. The user value of a userA 120A may be indicative of the overall value of the user A 120A to thecollaborative environment. The user value of the user A 120A mayincorporate the number of responses provided by the user A 120A relativeto the other users 120B-N, the number of ratings provided by the user A120A relative to the other users 120B-N, and the user response qualityscore of the user A 120A. The content providers 110A-N may use the uservalue of the users 120A-N to identify and reward the users who are themost valuable to the collaborative environment. The steps of determiningthe user value of the users 120A-N are discussed in more detail in FIG.6 below.

The content providers 110A-N may provide incentives, such as rewards, tothe users 120A-N, such as the user A 120A, if the user response qualityscore of the user A 120A, or the user value of the user A 120A, is abovea certain threshold. The rewards may motivate the users 120A-N toparticipate in the collaborative environment and provide accurateresponses to the collaborative environment. Alternatively or inaddition, the content providers 110A-N may eliminate a user A 120A fromthe collaborative environment if the user response quality score of theuser A 120A, or the user value of the user A 120A, falls below a certainthreshold. In the example of an organization, being eliminated from thecollaborative environment may be detrimental to the employment of a userA 120A, so the user A 120A may also be motivated to not fall below thethreshold. By properly incentivizing the users 120A-N, the contentproviders 110A-N may increase the accuracy of the collaborative review.

In operation one of the content providers 110A-N, such as the contentprovider A 110A may provide an item for review. The item may be aquestion whose answer is of value to the content provider A 110A. Thecontent provider A 110A may identify a period of time that the questionshould be provided to the users 120A-N for review. The content providerA 110A may also identify a set of the users 120A-N that the questionshould be provided to. The content provider A 110A may use the userresponse quality score of the users 120A-N, or the user value of theusers 120A-N, as a threshold for users 120A-N to be included in thereview. For example, the content provider A 110A may specify that onlythe users 120A-N with user response quality scores in the top tenpercent, or a user value in the top ten percent, should be provided theitem for review. The content provider A 110A may also select a set ofthe users 120A-N based on the demographics of the users 120A-N, orgenerally any characteristic of the users 120A-N capable of segmentingthe users 120A-N. The users 120A-N may be required to providedemographic information when they first register to participate in thecollaborative environment. In the case of an organization, the humanresources department of the organization may provide the demographicinformation of the users 120A-N.

The service provider 130 may provide the item to the users 120A-N forreview. The users 120A-N may be notified that the item is available,such as via an email notification. The users 120A-N may provide one ormore responses to the item. In the case of an initial item which is aquestion, the users 120A-N may provide one or more answers to thequestion. The service provider 130 may receive the responses from theusers 120A-N, and may provide the responses to the other users 120A-N.The users 120A-N may rate the responses.

Once the review period indicated by the content provider A 110A hasexpired, the service provider 130 may stop providing the item to theusers 120A-N. The service provider 130 may then calculate a responsequality score of each response provided by the users 120A-N. Asmentioned above, the response quality score may be determined bysubtracting the number of negative ratings received for the response bythe number of positive ratings received for the response and dividingthe result by the total number of unique users 120A-N who viewed theresponse. A response may only be eligible for receiving a responsequality score if the response satisfies the view threshold.Alternatively, the service provider 130 may calculate a total rating foreach response received from the users 120A-N. The total rating for aresponse may be a sum of each of the weighted ratings the responsereceived from the users 120A-N. A weighted rating may be equal to thevalue of the rating received from a user A 120A multiplied by the userresponse quality score of the user A 120A.

The service provider 130 may order the responses based on the responsequality score or total rating of each response. The service provider 130may provide the ordered list of responses to the content provider A 110Awho provided the initial item. The ordered list of responses may allowthe content provider A 110A to quickly and efficiently determine themost accurate response. The content provider A 110A may select one ormore responses as the most accurate response or responses. The contentprovider A 110A may provide an indication of the selection of the mostaccurate response or responses to the service provider 130.

At set intervals of time, such every three months, the service provider130 may determine which of the users 120A-N achieved a user responsequality score, or user value, above an incentive threshold. The users120A-N with a user response quality score, or user value, above thethreshold may be offered a reward. Alternatively or in addition theservice provider 130 may award the users 120A-N immediately when theiruser response quality score, or user value, reaches the incentivethreshold.

The service provider 130 may provide one or more reports to the contentproviders 110A-N and/or the users 120A-N indicating the activity of theusers 120A-N and/or the content providers 110A-N, such as displaying theuser response quality scores of the users 120A-N. The reports may alsoprovide information about the items reviewed in the collaborativeenvironment and the selected response for each initial item.

One or more of the users 120A-N and/or one or more of the contentproviders 110A-N may be an administrator of the collaborativeenvironment. An administrator may be generally responsible formaintaining the collaborative environment and may be responsible formaintaining the permissions of the users 120A-N and the contentproviders 110A-N. The administrator may need to approve of any new users120A-N added to the collaborative environment before the users 120A-Nare allowed to provide responses and ratings to the collaborativeenvironment.

FIG. 2 provides a view of a network environment 200 implementing thesystem of FIG. 1 or other systems for valuating users and user generatedcontent in a collaborative environment. Not all of the depictedcomponents may be required, however, and some implementations mayinclude additional components not shown in the figure. Variations in thearrangement and type of the components may be made without departingfrom the spirit or scope of the claims as set forth herein. Additional,different or fewer components may be provided.

The network environment 200 may include one or more web applications,standalone applications and mobile applications 210A-N, which may beclient applications of the content providers 110A-N. The networkenvironment 200 may also include one or more web applications,standalone applications, mobile applications 220A-N, which may be clientapplications of the users 120A-N. The web applications, standaloneapplications and mobile applications 210A-N, 220A-N, may collectively bereferred to as client applications 210A-N, 220A-N. The networkenvironment 200 may also include a network 230, a network 235, theservice provider server 240, a data store 245, and a third party server250.

Some or all of the service provider server 240 and third-party server250 may be in communication with each other by way of network 235. Thethird-party server 250 and service provider server 240 may eachrepresent multiple linked computing devices. Multiple distinct thirdparty servers, such as the third-party server 250, may be included inthe network environment 200. A portion or all of the third-party server250 may be a part of the service provider server 240.

The data store 245 may be operative to store data, such as userinformation, initial items, responses from the users 120A-N, ratings bythe users 120A-N, response quality scores, user response quality scores,user values, or generally any data that may need to be stored in a datastore 245. The data store 245 may include one or more relationaldatabases or other data stores that may be managed using various knowndatabase management techniques, such as SQL and object-based techniques.Alternatively or in addition the data store 245 may be implemented usingone or more of the magnetic, optical, solid state or tape drives. Thedata store 245 may be in direct communication with the service providerserver 240. Alternatively or in addition the data store 245 may be incommunication with the service provider server 240 through the network235.

The networks 230, 235 may include wide area networks (WAN), such as theinternet, local area networks (LAN), campus area networks, metropolitanarea networks, or any other networks that may allow for datacommunication. The network 230 may include the Internet and may includeall or part of network 235; network 235 may include all or part ofnetwork 230. The networks 230, 235 may be divided into sub-networks. Thesub-networks may allow access to all of the other components connectedto the networks 230, 235 in the system 200, or the sub-networks mayrestrict access between the components connected to the networks 230,235. The network 235 may be regarded as a public or private networkconnection and may include, for example, a virtual private network or anencryption or other security mechanism employed over the publicInternet.

The content providers 110A-N may use a web application 210A, standaloneapplication 210B, or a mobile application 210N, or any combinationthereof, to communicate to the service provider server 240, such as viathe networks 230, 235. Similarly, the users 120A-N may use a webapplication 220A, a standalone application 220B, or a mobile application220N to communicate to the service provider server 240, via the networks230, 235.

The service provider server 240 may provide user interfaces to thecontent providers 110A-N via the networks 230, 235. The user interfacesof the content providers 110A-N may be accessible through the webapplications, standalone applications or mobile applications 210A-N. Theservice provider server 240 may also provide user interfaces to theusers 120A-N via the networks 230, 235. The user interfaces of the users120A-N may also be accessible through the web applications, standaloneapplications or mobile applications 220A-N. The user interfaces may bedesigned using ADOBE FLEX. The user interfaces may be initiallydownloaded when the applications 210A-N, 220A-N first communicate withthe service provider server 240. The client applications 210A-N, 220A-Nmay download all of the code necessary to implement the user interfaces,but none of the actual data. The data may be downloaded from the serviceprovider server 240 as needed. The user interfaces may be developedusing the singleton development pattern, utilizing the model locatorfound within the cairngorm framework. Within the singleton pattern theremay be several data structures each with a corresponding data accessobject. The data structures may be structured to receive the informationfrom the service provider server 240.

The user interfaces of the content providers 110A-N may be operative toallow a content provider A 110A to provide an initial item, and allowthe content provider A 110A to specify a period of time for review ofthe item. The user interfaces of the users 120A-N may be operative todisplay the initial item to the users 120A-N, allow the users 120A-N toprovide responses and ratings, and display the responses and ratings tothe other users 120A-N. The user interfaces of the content providers110A-N may be further operative to display the ordered list of responsesto the content provider A 110A and allow the content provider to providean indication of the selected response.

The web applications, standalone applications and mobile applications210A-N, 220A-N may be connected to the network 230 in any configurationthat supports data transfer. This may include a data connection to thenetwork 230 that may be wired or wireless. The web applications 210A,220A may run on any platform that supports web content, such as a webbrowser or a computer, a mobile phone, personal digital assistant (PDA),pager, network-enabled television, digital video recorder, such asTIVO®, automobile and/or any appliance capable of data communications.

The standalone applications 210B, 220B may run on a machine that mayhave a processor, memory, a display, a user interface and acommunication interface. The processor may be operatively connected tothe memory, display and the interfaces and may perform tasks at therequest of the standalone applications 210B, 220B or the underlyingoperating system. The memory may be capable of storing data. The displaymay be operatively connected to the memory and the processor and may becapable of displaying information to the content provider B 110B or theuser B 120B. The user interface may be operatively connected to thememory, the processor, and the display and may be capable of interactingwith a user B 120B or a content provider B 110B. The communicationinterface may be operatively connected to the memory, and the processor,and may be capable of communicating through the networks 230, 235 withthe service provider server 240, and the third party server 250. Thestandalone applications 210B, 220B may be programmed in any programminglanguage that supports communication protocols. These languages mayinclude: SUN JAVA %, C++, C#, ASP, SUN JAVASCRIPT®, asynchronous SUNJAVASCRIPT®, or ADOBE FLASH ACTIONSCRIPT®, ADOBE FLEX, and PHP, amongstothers.

The mobile applications 210N, 220N may run on any mobile device that mayhave a data connection. The data connection may be a cellularconnection, a wireless data connection, an internet connection, aninfra-red connection, a Bluetooth connection, or any other connectioncapable of transmitting data.

The service provider server 240 may include one or more of thefollowing: an application server, a data store, such as the data store245, a database server, and a middleware server. The application servermay be a dynamic HTML server, such as using ASP, JSP, PHP, or othertechnologies. The service provider server 240 may co-exist on onemachine or may be running in a distributed configuration on one or moremachines. The service provider server 240 may collectively be referredto as the server. The service provider server 240 may implement a serverside wiki engine, such as ATLASSIAN CONFLUENCE. The service providerserver 240 may receive requests from the users 120A-N and the contentproviders 110A-N and may provide data to the users 120A-N and thecontent providers 110A-N based on their requests. The service providerserver 240 may communicate with the client applications 210A-N, 220A-Nusing extensible markup language (XML) messages.

The third party server 250 may include one or more of the following: anapplication server, a data source, such as a database server, and amiddleware server. The third party server may implement any third partyapplication that may be used in a system for valuating users and usergenerated content in a collaborative environment, such as a userverification system. The third party server 250 may co-exist on onemachine or may be running in a distributed configuration on one or moremachines. The third party server 250 may receive requests from the users120A-N and the content providers 110A-N and may provide data to theusers 120A-N and the content providers 110A-N based on their requests.

The service provider server 240 and the third party server 250 may beone or more computing devices of various kinds, such as the computingdevice in FIG. 11. Such computing devices may generally include anydevice that may be configured to perform computation and that may becapable of sending and receiving data communications by way of one ormore wired and/or wireless communication interfaces. Such devices may beconfigured to communicate in accordance with any of a variety of networkprotocols, including but not limited to protocols within theTransmission Control Protocol/Internet Protocol (TCP/IP) protocol suite.For example, the web applications 210A, 210A may employ HTTP to requestinformation, such as a web page, from a web server, which may be aprocess executing on the service provider server 240 or the third-partyserver 250.

There may be several configurations of database servers, such as thedata store 245, application servers, and middleware servers included inthe service provider server 240, or the third party server 250. Databaseservers may include MICROSOFT SQL SERVER®, ORACLE®, IBM DB2® or anyother database software, relational or otherwise. The application servermay be APACHE TOMCAT®, MICROSOFT IIS®, ADOBE COLDFUSION®, or any otherapplication server that supports communication protocols. The middlewareserver may be any middleware that connects software components orapplications.

The networks 230, 235 may be configured to couple one computing deviceto another computing device to enable communication of data between thedevices. The networks 230, 235 may generally be enabled to employ anyform of machine-readable media for communicating information from onedevice to another. Each of networks 230, 235 may include one or more ofa wireless network, a wired network, a local area network (LAN), a widearea network (WAN), a direct connection such as through a UniversalSerial Bus (USB) port, and the like, and may include the set ofinterconnected networks that make up the Internet. The networks 230, 235may include any communication method by which information may travelbetween computing devices.

In operation the client applications 210A-N, 220A-N may make requestsback to the service provider server 240. The service provider server 240may access the data store 245 and retrieve information in accordancewith the request. The information may be formatted as XML andcommunicated to the client applications 210A-N, 220A-N. The clientapplications 210A-N, 220A-N may display the XML appropriately to theusers 120A-N, and/or the content providers 110A-N.

FIG. 3 provides a view of the server-side components in a networkenvironment 300 implementing the system of FIG. 2 or other systems forvaluating users and user generated content in a collaborativeenvironment. Not all of the depicted components may be required,however, and some implementations may include additional components notshown in the figure. Variations in the arrangement and type of thecomponents may be made without departing from the spirit or scope of theclaims as set forth herein. Additional, different or fewer componentsmay be provided.

The network environment 300 may include the network 235, the serviceprovider server 240, and the data store 245. The server provider server240 may include an interface 310, a response processor 320, a ratingprocessor 330, rating calculator 340, a user quality score calculator350, and a user value calculator 360. The interface 310, responseprocessor 320, rating processor 330, rating calculator 340, the userquality score calculator 350, and the user value calculator 360 may beprocesses running on the service provider server 240, may be hardwarecomponents of the service provider server 240, such as dedicatedprocessors or dedicated processing cores, or may be separate computingdevices, such as the one described in FIG. 11.

The interface 310 may communicate with the users 120A-N and the contentproviders 110A-N via the networks 230, 235. The response processor 320may process responses and initial items from the users 120A-N and thecontent providers 110A-N, the rating processor 330 may process ratingsreceived from the users 120A-N, views of responses of the users 120A-N,and selections of the content provider A 110A, the rating calculator 340may calculate the response quality score, weighted ratings and totalratings of the responses, the user response quality score calculator 350may calculate the user response quality scores of the users 120A-N, andthe user value calculator may determine the user values of the users120A-N.

In operation the interface 310 may receive data from the contentproviders 110A-N or the users 120A-N via the network 235. For example,one of the content providers 110A-N, such as the content provider A110A, may provide an initial item, and one of the users 120A-N, such asthe user A 120A may provide a response or a rating of a response. In thecase of an initial item received from the content provider A 110A, theinterface 310 may communicate the initial item to the response processor320. The response processor 320 may store the initial item in the datastore 245. The response processor 320 may store data describing thecontent provider A 110A who provided the initial item and the date/timethe initial item was provided. The response processor 320 may also storethe review period identified by the content provider A 110A for theitem.

In the case of a response received from the user A 120A, the interface310 may communicate the response to the response processor 320. Theresponse processor 320 may store the response in the data store 245along with the initial item the response was based on. The responseprocessor 320 may store user data describing the user A 120A whoprovided the response and the date/time the response was provided. Inthe case of a rating received from the user A 120A, the interface 310may communicate the rating to the rating processor 330. The ratingprocessor 330 may store the rating in the data store 245 along with theresponse the rating was given for. The rating processor 330 may alsostore user data describing the user A 120A who provided the rating, userdata describing the user B 120B who provided the response that wasrated, and the date/time the response was rated.

The rating processor 330 may also store data when one of the users120A-N views a response of the user A 120A. The interface 310 mayreceive an indication that a response of the user A 120A was viewed bythe user B 120B and may communicate the indication to the ratingprocessor 330. The rating processor 330 may store user data describingthe user A 120A who provided the response, user data describing the userB 120B who viewed the response, the response viewed, and the date/timethe response was viewed.

The rating processor 330 may also store the response selected by thecontent provider A 110A as the most accurate response. The interface 310may receive an indication of the response selected by the contentprovider A 110A via the interface 310. The interface 310 may communicatethe indication of the selected response to the rating processor 330. Therating processor 330 may store the selected response, user datadescribing the user A 120A who provided the selected response, user datadescribing the content provider A 110A, and the date/time the selectedresponse was received by the interface 310.

The rating calculator 340 may determine the response quality score ofresponses to an initial item, and may order the responses based on theirresponse quality scores. The rating calculator 340 may follow the stepsof FIG. 4 to determine the response quality score of a response.Alternatively or in addition, the rating calculator 340 may calculatethe weighted ratings and the total ratings of the responses, and mayorder the responses based on their total ratings. The rating calculator340 may follow the steps of FIG. 8 to determine the weighted ratings andthe total ratings of the responses.

Once the rating calculator 340 has calculated the response qualityscores and/or total rating of each response, the rating calculator 340may order the responses based on the response quality scores and/orratings and may provide the ordered responses, with the response qualityscores and/or total ratings, to the content provider A 110A who providedthe initial item.

The service provider server 240 may re-calculate a user response qualityscore of a user A 120A each time the underlying data the score is basedon changes. Alternatively or in addition the rating calculator 340 mayrequest the user response quality scores of the users 120A-N when therating calculator 340 calculates the total rating of each response atthe end of review period.

The user response quality score calculator 350 may receive a request forthe user response quality score of the user A 120A. The user responsequality score calculator 350 may retrieve one or more values from thedata store 245 in order to determine the user response quality score ofthe user A 120A, such as the ratings of the responses provided by theuser A 120A. The user response quality score calculator 350 may followthe steps of FIG. 5, FIG. 7, or FIG. 8 to determine the user responsequality score of the user A 120A.

The user value calculator 360 may determine the user values of the users120A-N at specified intervals, or when requested, such as by one of thecontent providers 110A-N. The user value calculator 360 may retrieve oneor more values from the data store 245 in order to calculate the uservalue, such as the number of contributions provided by the user A 120A,the number of ratings provided by the user A 120A, and the user responsequality score of the user A 120A. The steps of determining the uservalues of the users 120A-N are discussed in more detail in FIG. 5 below.

FIG. 4 is a flowchart illustrating the operations of determiningresponse quality scores in the systems of FIG. 1, FIG. 2, or FIG. 3, orother systems for valuating users and user generated content in acollaborative environment. The steps of FIG. 4 are described as beingperformed by the service provider server 240. However, the steps may beperformed by the operating system of the service provider server 240, aprocessor of the service provider server 240, a processing core of theservice provider server 240, any other hardware component of the serviceprovider server 240, or any combination thereof. Alternatively the stepsmay be performed by an external hardware component, an external softwareprocess, or any combination thereof.

At step 405, the service provider server 240 may retrieve one or moreresponses received from the users 120A-N, such as from the data store245. The responses may have been provided by the users 120A-N inresponse to an initial item provided by one of the content providers110A-N. At step 410, the service provider server 240 may determine thenumber of unique users who viewed each response. At step 420, theservice provider server 240 may select the first response from the setof retrieved responses. At step 425, the service provider server 240determines whether the selected response satisfies the view threshold.The view threshold may indicate the minimum number of unique users120A-N who must view a response in order for the response to be eligibleto receive a response quality score. The view threshold may bedetermined by an administrator, or the view threshold may have a defaultvalue, such as only responses in the top two-thirds of responses interms of total unique views satisfy the view threshold.

If, at step 425, the service provider server 240 determines that theselected response satisfies the view threshold, the service providerserver 240 moves to step 430. At step 430, the service provider server240 determines the net rating of the response. For example, if the users120A-N rate the responses with a positive rating or a negative rating,the net rating may be the total number of negative ratings the responsereceived subtracted from the total number of positive ratings theresponse received. Alternatively or in addition the response may berated on a scale of one to ten. In this example, the service providerserver 240 may determine the net rating of the response by determiningthe sum of all of the ratings.

At step 440, the service provider server 240 may determine the responsequality score by dividing the net rating by the total number of uniqueusers 120A-N who viewed the response. At step 450, the service providerserver 240 may store the response quality score of the response in thedata store 245. The service provider server 245 may also store anassociation between the response quality score and the response suchthat the response quality score can be retrieved based on the response.At step 455, the service provider server 240 may determine whether thereare any additional responses for the initial item which have yet to beevaluated for satisfying the view threshold. If, at step 455, theservice provider server 240 determines that there are additionalresponses, the service provider server 240 moves to step 460. At step460, the service provider server 240 may select the next response fromthe set of responses and repeats steps 425-455 for the next response.If, at step 425, the service provider server 240 determines that theselected response does not satisfy the view threshold, the serviceprovider server 240 may move to step 455 and may determine whether anyother responses have not yet been evaluated for satisfying the viewthreshold.

If, at step 455, the service provider server 240 determines that all ofthe responses have been evaluated for satisfying the view threshold, theservice provider 240 may move to step 470. At step 470, the serviceprovider server 240 may retrieve the response quality scores andassociated responses from the data store 245. At step 480, the serviceprovider server 240 may transform the response quality scores andresponses into a graphical representation. FIG. 10 may provide anexample of a graphical representation of responses and response qualityscores. At step 490, the service provider server 240 may provide thegraphical representation to the content provider A 110A who provided theinitial item the responses relate to, such as through a device of theuser. For example, the service provider server 240 may provide thegraphical representation to a content provider A 110A, or to anadministrator.

FIG. 5 is a flowchart illustrating the operations of determining a userresponse quality score in the systems of FIG. 1, FIG. 2, or FIG. 3, orother systems for valuating users and user generated content in acollaborative environment. The steps of FIG. 5 are described as beingperformed by the service provider server 240. However, the steps may beperformed by the operating system of the service provider server 240, aprocessor of the service provider server 240, a processing core of theservice provider server 240, any other hardware component of the serviceprovider server 240, or any combination thereof. Alternatively the stepsmay be performed by an external hardware component, an external softwareprocess, or any combination thereof.

At step 510, the service provider server 240 identifies the set of users120A-N of the collaborative environment. For example, the serviceprovider server 240 may retrieve user data describing the users 120A-Nfrom the data store 245. At step 520, the service provider server 240may select the first user from the set of users 120A-N of thecollaborative environment. At step 525, the service provider server 240may determine whether the selected user satisfies the contributionthreshold. The contribution threshold may indicate the minimum number ofresponses a user A 120A should contribute to the collaborativeenvironment before the user A 120A is eligible to receive a userresponse quality score. The contribution threshold may be determined byan administrator or may have a default value. For example, a defaultcontribution threshold may indicate that only the users 120A-N in thetop two-thirds of the users 120A-N in terms of contributions to thecollaborative environment satisfy the contribution threshold.

If, at step 525, the service provider server 240 determines that theselected user satisfies the contribution threshold, the service providerserver 240 moves to step 530. At step 530, the service provider serverretrieves the response quality scores of all of the responses providedby the selected user. At step 535, the service provider server 240determines the user response quality score of the selected user bydetermining the average of the response quality scores of the responsesprovide by the selected user. At step 540, the service provider server240 stores the user response quality score of the selected user in thedata store 245. The service provider server 240 may also store anassociation between the user response quality score and the user datasuch that the user response quality score can be retrieved based on theuser data.

At step 545, the service provider server 240 determines whether thereare any additional users 120B-N which have yet to be evaluated againstthe contribution threshold. If, at step 545, the service provider server240 determines there are additional users, the service provider server240 moves to step 550. At step 550, the service provider server 240selects the next user and repeats steps 525-545 for the next user. If,at step 525, the service provider server 240 determines that theselected user does not satisfy the contribution threshold, the serviceprovider server 240 moves to step 545. Once the service provider server240 have evaluated all of the users 120A-N against the contributionthreshold, and determined user response quality scores for eligibleusers 120A-N, the service provider server 240 moves to step 560.

At step 560, the service provider server 240 retrieves the determineduser response quality scores, and the associated user data from the datastore 245. At step 570, the service provider server 240 transforms theuser response quality scores and the associated user data into agraphical representation. The graphical representation may be similar tothe graphical representation illustrated in FIG. 10, but displaying userdata and user response quality scores. At step 580, the service providerserver 240 provides the graphical representation to a user, such asthrough a device of the user. For example, the service provider server240 may provide the graphical representation to one of the contentproviders 110A-N or to an administrator.

FIG. 6 is a flowchart illustrating the operations of determining thevalue of users in the systems of FIG. 1, FIG. 2, or FIG. 3, or othersystems for valuating users and user generated content in acollaborative environment. The steps of FIG. 6 are described as beingperformed by the service provider server 240. However, the steps may beperformed by the operating system of the service provider server 240, aprocessor of the service provider server 240, a processing core of theservice provider server 240, any other hardware component of the serviceprovider server 240, or any combination thereof. Alternatively the stepsmay be performed by an external hardware component, an external softwareprocess, or any combination thereof.

At step 605, the service provider server 240 may identify a user A 120Afor which to determine a user value. For example, one of the contentproviders 110A-N, or an administrator, may request the user value of auser A 120A. Alternatively, the service provider server 240 maydetermine the user values of all of the users 120A-N, such as togenerate a report or graphical representation of user values. At step610, the service provider server 240 may determine the contribution rankof the user A 120A. The contribution rank may identify the number ofcontributions to the collaborative environment, such as responses,provided by the user A 120A relative to the number of contributionsprovided by the other users 120B-N. For example, if the user A 120Aprovided the most responses to the collaborative environment, the user A120A may have a contribution rank of 1.

At step 620, the service provider server 240 may determine the ratingrank of the user A 120A. The rating rank may identify the number ofresponses which were rated by the user A 120A relative to the number ofresponses rated by the other users 120B-N. For example, if the user A120A rated the most responses in the collaborative environment, the userA 120A may have a rating rank of 1. At step 630, the service providerserver 240 determines the user response quality rank of the user A 120A.The user response quality rank of the user A 120A may represent the userresponse quality score of the user A 120A as it compares to the userresponse quality scores of the other users 120B-N. For example, if theuser A 120A received the highest user response quality score out of theusers 120A-N, the user A 120A may receive a user response quality rankof 1.

At step 640, the service provider server 240 inputs the contributionrank, rating rank, and user response quality rank of the user A 120A toan algorithm to determine the user value of the user A 120A. Thealgorithm may apply different weights to each of the ranks. The weightsapplied to the ranks may vary by implementation, but may generallyrepresent the amount of value an organization wishes to accord toproviding response, rating responses and providing quality responses inthe collaborative environment. In one example, a weight of 0.25 may beapplied to the contribution rank and the rating rank, and a weight of0.5 may be applied to the user response quality rank. In this example,an organization may wish to weigh the quality of the responses of theusers 120A-N more than the number of responses provided or the number ofresponses rated by the users 120A-N. Thus, in this example theorganization may wish to weigh quality over quantity. At step 650, theservice provider server 240 stores the user value in the data store 245.The service provider server 240 may store an association between theuser data of the user A 120A and the user value such that the user valuecan be retrieved based on the user data.

FIG. 7 is a flowchart illustrating the alternative operations ofdetermining a user response quality score in the systems of FIG. 1, FIG.2, or FIG. 3, or other systems for valuating users and user generatedcontent in a collaborative environment. The steps of FIG. 7 aredescribed as being performed by the service provider server 240.However, the steps may be performed by the operating system of theservice provider server 240, a processor of the service provider server240, a processing core of the service provider server 240, any otherhardware component of the service provider server 240, or anycombination thereof. Alternatively the steps may be performed by anexternal hardware component, an external software process, or anycombination thereof.

At step 710, the service provider server 240 may receive a request for auser response quality score, such as from the rating calculator 340. Theuser response quality score may be requested during the calculation of aweighted score of a user A 120A. At step 720, the service providerserver 240 may retrieve, from the data store 245, the number ofresponses the user A 120A provided to the service provider server 240.At step 730, the service provider server 240 may retrieve, from the datastore 245, the number of times the responses of the user A 120A wereviewed by other users 120B-N. At step 740, the service provider server240 may retrieve, from the data store 245, the total rating of each ofthe responses provided by the user A 120A. The total rating of each ofthe responses provided by the user A 120A may be used to determine theaverage total rating of the responses of the user A 120A. At step 750,the service provider server 240 may retrieve, from the data store 245,the number of responses of the user A 120A selected by one of thecontent providers 110A-N.

At step 760, the service provider server 240 may use the retrieved datato calculate the user response quality score of the user A 120A. Forexample, the service provider server 240 may determine the normalizedvalue of each of the individual metrics. A normalized value may bedetermined by calculating the average value of a given metric for all ofthe users 120A-N, and dividing the value of the user A 120A by theaverage value of all the users 120A-N. The service provider server 240may then add the normalized values of the user A 120A to determine a sumof the normalized values. The sum of the normalized values may be theuser response quality score of the user A 120A. Alternatively or inaddition, the sum of the normalized values may be normalized to obtainthe user response quality score. Alternatively or in addition theservice provider server 240 may add all of the individual valuestogether and normalize the sum of the individual values. Alternativelyor in addition the service provider server 240 may weight one of thevalues more than the others, such as weighting the average rating of theresponses of the user A 120A. Alternatively or in addition the userresponse quality score may be equal to one of the metrics, such as thenumber of responses provided by the user A 120A to the service providerserver 240.

Alternatively or in addition the service provider server 240 maycalculate a normalized value for each metric by determining the maximumvalue of the metric for all of the users 120A-N, and dividing the valueof the user A 120A by the maximum value of all the users 120A-N. Forexample, the service provider server 240 may use three metrics indetermining the user response quality score value: the number ofresponses the user A 120A provided to the service provider server 240,the number of times the responses of the user A 120A were viewed by theusers 120A-N, and the average total rating the responses of the user A120A received. The user service provider server 240 may determine themaximum number of responses provided by any of the users 120A-N, themaximum number of times responses of any of the users 120A-N were viewedby the other users 120A-N, and the maximum average total rating theresponses of any of the users 120A-N received.

The service provider server 240 may calculate the normalized value ofeach of the metrics by dividing the value of the user A 120A by themaximum value received by the service provider server 240 for themetric. For example the normalized number of responses provided by theuser A 120A may be calculated by dividing the number of responsesprovided by the user A 120A by the maximum number of responses receivedby any of the users 120A-N. Once the normalized values are determined,the service provider server 240 may multiply the nominal values by aweight. The weight may be indicative of the importance of the metric tothe total rating of the response. For example, the service providerserver 240 may multiply the normalized number of responses by 0.16, thenormalized number of views by 0.33, and the normalized average responsetotal rating by 0.5. After multiplying the normalized values by aweight, the service provider server 240 may add together the results todetermine the user response quality score of the user A 120A. At step770, the service provider server 240 may provide the user responsequality score to the requestor, such as the rating calculator 330.

FIG. 8 is a flowchart illustrating alternative operations of determiningand maintaining a user response quality score in the systems of FIG. 1,FIG. 2, or FIG. 3, or other systems for valuating users and usergenerated content in a collaborative environment. The steps of FIG. 8are described as being performed by the service provider server 240.However, the steps may be performed by the operating system of theservice provider server 240, a processor of the service provider server240, a processing core of the service provider server 240, any otherhardware component of the service provider server 240, or anycombination thereof. Alternatively the steps may be performed by anexternal hardware component, an external software process, or anycombination thereof.

At step 805, one of the users 120A-N, such as the user A 120A mayinteract with the service provider server 240 for the first time, suchas navigating to web login page of the service provider server 240. Theuser A 120A may be required to provide information to create an accountwith the service provider server 240 in order to participate in thecollaborative environment. The information may include personalinformation, such as name, home address, email address, telephonenumber, or generally any personal information, demographic information,such as age, ethnicity, gender, or generally any information that may beused by the service provider server 240. The user A 120A may be grantedimmediate access to the service provider server 240, or an administratorof the service provider server 240 may have to approve of the user A120A before the user A 120A is granted access to the service providerserver 240.

At step 810, the service provider server 240 may calculate an initialuser response quality score of the user A 120A. The initial userresponse quality score may be 0, may be a default score, may be a scorespecified by an administrator with knowledge of the user A 120A, or maybe determined based on the information the user A 120A provided to theservice provider server 240. At steps 815-830, the service providerserver 240 may continually check for updates to the values that the userresponse quality score may be based on. Alternatively or in addition theuser response quality score may only be calculated when a weightedrating of the user A 120A is being determined, or at the end of a reviewperiod.

At step 815, the service provider server 240 may determine whether theuser A 120A provided a response to the service provider server 240. Ifthe user A 120A did not provide a response to the service providerserver 240, the service provider server 240 may move to step 820. Atstep 820, the service provider server 240 may determine whether aresponse of the user A 120A was viewed by one of the other users 120B-N.If a response of the user A 120A was not viewed by one of the otherusers 120B-N, the service provider server 240 may move to step 825. Atstep 825, the service provider server 240 determines whether a responseof the user A 120A was rated by one of the other users 120B-N. If theresponse of the user A 120A was not rated by one of the other users120B-N, the service provider server 240 may move to step 830. At step830, the service provider server 240 determines whether a response ofthe user A 120A was selected by one of the content providers 110A-N asthe most accurate response. If a response of the user A 120A was notselected by one of the content providers 110A-N, the service providerserver 240 may return to step 815 and continue to check for updates.

If at steps 815-830 the user A 120A provides a response, or a responseof the user A 120A is viewed by one of the other users 120B-N, or aresponse of the user A 120A is rated by one of the other users 120B-N,or a response of the user A 120A is selected by one of the contentproviders 110A-N, the service provider server 240 may move to step 835.Alternatively or in addition, if any other values are used to determinethe user response quality score, a change to one of those values maycause the service provider server 240 to move to step 835.

At step 835, the service provider server 240 may re-calculate the userresponse quality score of the user A 120A based on the changes in therelevant values. The operations of calculating the user response qualityscore are discussed in more detail in FIG. 5 and FIG. 7. At step 840,the service provider server 240 may determine whether the re-calculateduser response quality score of the user A 120A is above the incentivethreshold. If the user response quality score of the user A 120A isabove the incentive threshold, the service provider server 240 may moveto step 850. At step 850, the service provider server 240 may providethe user A 120A with the incentive, such as a gift certificate. Theservice provider server 240 may then return to step 815 and repeat thechecking process. If, at step 840, the user response quality score isnot above the incentive threshold the service provider server 240 maymove to step 845. At step 845, the service provider server 240 maynotify the user A 120A that their user response quality score haschanged, but is still below the incentive threshold. The serviceprovider server 240 may provide the user A 120A with the number ofpoints their user response quality score must be raised in order toreach the incentive threshold. The service provider server 240 may thenmove to step 815 and continue to check for updates.

Alternatively or in addition the service provider server 240 maymaintain multiple incentive threshold tiers, such as a bronze tier, asilver tier, and a gold tier. The users 120A-N may be rewarded with morevaluable incentives when their user response quality score reaches ahigher tier. For example, the gold tier may be users 120A-N with a userresponse quality score in the top ten percent of the users 120A-N, thesilver tier may be the top twenty percent and the bronze tier may be thetop thirty percent. The gold tier may have the best rewards, while thesilver tier may be middle level rewards and the bronze tier may be lowerlevel rewards.

Alternatively or in addition the service provider server 240 maymaintain a lower user response quality score threshold. If the userresponse quality score of a user A 120A falls below the lower userresponse quality score threshold, the user A 120A may be warned thattheir user response quality score is too low. Alternatively or inaddition if the user response quality score of a user A 120A falls belowthe lower threshold the user A 120A may be removed from the serviceprovider server 240. Alternatively or in addition, in the case of anorganization, if the user response quality score of a user A 120A fallsbelow the lower threshold, the user A 120A may be terminated from theorganization.

Alternatively or in addition the service provider server 240 may use theuser values to determine whether to reward the users 120A-N. Forexample, the service provider server 240 may reward the users 120A-N ifthe user value of the users 120A-N is above the incentive threshold.FIG. 6 above describes the steps of determining the user values of theusers 120A-N.

FIG. 9 is a flowchart illustrating the operations of weighting userratings with user response quality scores in the systems of FIG. 1, FIG.2, or FIG. 3, or other systems for valuating users and user generatedcontent in a collaborative environment. The steps of FIG. 9 aredescribed as being performed by the service provider server 240.However, the steps may be performed by the operating system of theservice provider server 240, a processor of the service provider server240, a processing core of the service provider server 240, any otherhardware component of the service provider server 240, or anycombination thereof. Alternatively the steps may be performed by anexternal hardware component, an external software process, or anycombination thereof.

At step 910, the service provider server 240 may receive an initial itemfrom the content provider A 110A. The content provider A 110A mayprovide any item which may be commented on, or responded to, such as aquestion, an image, an audio clip, a news article, or a video. Thecontent provider A 110A may also provide a period of time that the itemshould be available for review by the users 120A-N, such as one week.Alternatively or in addition the content provider A 110A may selectwhich of the users 120A-N should be able to review the item. The contentprovider A 110A may only want a subset of the users 120A-N to review theitem, such as the users 120A-N who have the highest user responsequality scores.

At step 920, the service provider server 240 may receive responses fromthe users 120A-N to the initial item. For example, if the initial itemis a question the users 120A-N may respond with answers to the question.At step 930, the service provider server 240 may receive ratings of theresponses from the users 120-N. For example, the users 120A-N mayprovide ratings indicating whether they believe a given response isaccurate for the initial item.

At step 940, the review period for the initial item may have ended, andthe service provider server 240 may calculate the user response qualityscores of the users 120A-N who provided ratings. The operations ofcalculating the user response quality score are discussed in more detailin FIG. 5 and FIG. 7. At step 950, the service provider server 240 maydetermine the weighted rating of each rating provided by the users120A-N. For each rating received, the weighted rating may be calculatedby multiplying the user response quality score of the user A 120A whoprovided the rating by the value of the rating. At step 960, the serviceprovider server 240 may determine the total rating of each responsebased on the weighted ratings of each response. For example the totalrating of each response may be calculated by determining the averageweighted rating of each response.

At step 970, the service provider server 240 may provide the orderedlist of responses to the content provider A 110A. The responses may beordered based on the total ratings of the responses. At step 980, theservice provider server 240 may receive an indication of the responseselected by the content provider A 110A as the most accurate response.

FIG. 10 is a screenshot of a reporting screen 1000 in the systems ofFIG. 1, FIG. 2, or FIG. 3, or other systems for valuating users and usergenerated content in a collaborative environment. The reporting screen1000 may include a report subsection 1010, and an initial itemsubsection 1020. The report subsection 1010 may include one or moreresponses 1018, or ideas, and each response 1018 may be associated witha response quality score 1018. The report subsection 1010 may alsodisplay the number of positive ratings each response 1018 received, thenumber of negative ratings each response 1018 received, and the numberof users 120A-N who viewed each response 1018.

The initial item subsection 1020 may include an item creation subsection1024, an item title 1026, and an item description 1022. The item title1026 may display the title of the initial item for which the responses1018 were submitted. The item creation subsection 1024 may display oneor more data items relating to the creation of the initial item, such asthe user A 120A who submitted the item and the date the item wassubmitted on. The item description subsection 1022 may display adescription of the initial item.

In operation, an administrator may view the report subsection 1010 toview the responses 1018 which received the highest response qualityscores 1016. The administrator may view the initial item associated withthe responses 1018 in the initial idea subsection 1020. The responsequality scores 1016 may be transformed into a graphical representationto allow the administrator to quickly identify the highest responsequality scores 1016. For example, the response quality scores 1016 maybe enclosed in a graphic of a box. The shading of the graphic maycorrelate to the response quality score 1016 such that higher scoreshave a lighter shading than lower scores. Alternatively or in additionthe graphical representations of the response quality scores 1016 maydiffer by size, color, shape, or generally any graphical attribute inorder to allow an administrator to quickly identify the responses withthe highest response quality score.

FIG. 11 illustrates a general computer system 1100, which may representa service provider server 240, a third party server 250, the clientapplications 210A-N, 220A-N, or any of the other computing devicesreferenced herein. The computer system 1100 may include a set ofinstructions 1124 that may be executed to cause the computer system 1100to perform any one or more of the methods or computer based functionsdisclosed herein. The computer system 1100 may operate as a standalonedevice or may be connected, e.g., using a network, to other computersystems or peripheral devices.

In a networked deployment, the computer system may operate in thecapacity of a server or as a client user computer in a server-clientuser network environment, or as a peer computer system in a peer-to-peer(or distributed) network environment. The computer system 1100 may alsobe implemented as or incorporated into various devices, such as apersonal computer (PC), a tablet PC, a set-top box (STB), a personaldigital assistant (PDA), a mobile device, a palmtop computer, a laptopcomputer, a desktop computer, a communications device, a wirelesstelephone, a land-line telephone, a control system, a camera, a scanner,a facsimile machine, a printer, a pager, a personal trusted device, aweb appliance, a network router, switch or bridge, or any other machinecapable of executing a set of instructions 1124 (sequential orotherwise) that specify actions to be taken by that machine. In aparticular embodiment, the computer system 1100 may be implemented usingelectronic devices that provide voice, video or data communication.Further, while a single computer system 1100 may be illustrated, theterm “system” shall also be taken to include any collection of systemsor sub-systems that individually or jointly execute a set, or multiplesets, of instructions to perform one or more computer functions.

As illustrated in FIG. 11, the computer system 1100 may include aprocessor 1102, such as, a central processing unit (CPU), a graphicsprocessing unit (GPU), or both. The processor 1102 may be a component ina variety of systems. For example, the processor 1102 may be part of astandard personal computer or a workstation. The processor 1102 may beone or more general processors, digital signal processors, applicationspecific integrated circuits, field programmable gate arrays, servers,networks, digital circuits, analog circuits, combinations thereof, orother now known or later developed devices for analyzing and processingdata. The processor 1102 may implement a software program, such as codegenerated manually (i.e., programmed).

The computer system 1100 may include a memory 1104 that can communicatevia a bus 1108. The memory 1104 may be a main memory, a static memory,or a dynamic memory. The memory 1104 may include, but may not be limitedto computer readable storage media such as various types of volatile andnon-volatile storage media, including but not limited to random accessmemory, read-only memory, programmable read-only memory, electricallyprogrammable read-only memory, electrically erasable read-only memory,flash memory, magnetic tape or disk, optical media and the like. In onecase, the memory 1104 may include a cache or random access memory forthe processor 1102. Alternatively or in addition, the memory 1104 may beseparate from the processor 1102, such as a cache memory of a processor,the system memory, or other memory. The memory 1104 may be an externalstorage device or database for storing data. Examples may include a harddrive, compact disc (“CD”), digital video disc (“DVD”), memory card,memory stick, floppy disc, universal serial bus (“USB”) memory device,or any other device operative to store data. The memory 1104 may beoperable to store instructions 1124 executable by the processor 1102.The functions, acts or tasks illustrated in the figures or describedherein may be performed by the programmed processor 1102 executing theinstructions 1124 stored in the memory 1104. The functions, acts ortasks may be independent of the particular type of instructions set,storage media, processor or processing strategy and may be performed bysoftware, hardware, integrated circuits, firm-ware, micro-code and thelike, operating alone or in combination. Likewise, processing strategiesmay include multiprocessing, multitasking, parallel processing and thelike.

The computer system 1100 may further include a display 1114, such as aliquid crystal display (LCD), an organic light emitting diode (OLED), aflat panel display, a solid state display, a cathode ray tube (CRT), aprojector, a printer or other now known or later developed displaydevice for outputting determined information. The display 1114 may actas an interface for the user to see the functioning of the processor1102, or specifically as an interface with the software stored in thememory 1104 or in the drive unit 1106.

Additionally, the computer system 1100 may include an input device 1112configured to allow a user to interact with any of the components ofsystem 1100. The input device 1112 may be a number pad, a keyboard, or acursor control device, such as a mouse, or a joystick, touch screendisplay, remote control or any other device operative to interact withthe system 1100.

The computer system 1100 may also include a disk or optical drive unit1106. The disk drive unit 1106 may include a computer-readable medium1122 in which one or more sets of instructions 1124, e.g. software, canbe embedded. Further, the instructions 1124 may perform one or more ofthe methods or logic as described herein. The instructions 1124 mayreside completely, or at least partially, within the memory 1104 and/orwithin the processor 1102 during execution by the computer system 1100.The memory 1104 and the processor 1102 also may includecomputer-readable media as discussed above.

The present disclosure contemplates a computer-readable medium 1122 thatincludes instructions 1124 or receives and executes instructions 1124responsive to a propagated signal; so that a device connected to anetwork 235 may communicate voice, video, audio, images or any otherdata over the network 235. Further, the instructions 1124 may betransmitted or received over the network 235 via a communicationinterface 1118. The communication interface 1118 may be a part of theprocessor 1102 or may be a separate component. The communicationinterface 1118 may be created in software or may be a physicalconnection in hardware. The communication interface 1118 may beconfigured to connect with a network 235, external media, the display1114, or any other components in system 1100, or combinations thereof.The connection with the network 235 may be a physical connection, suchas a wired Ethernet connection or may be established wirelessly asdiscussed below. Likewise, the additional connections with othercomponents of the system 1100 may be physical connections or may beestablished wirelessly. In the case of a service provider server 240 orthe content provider servers 110A-N, the servers may communicate withusers 120A-N through the communication interface 1118.

The network 235 may include wired networks, wireless networks, orcombinations thereof. The wireless network may be a cellular telephonenetwork, an 802.11, 802.16, 802.20, or WiMax network. Further, thenetwork 235 may be a public network, such as the Internet, a privatenetwork, such as an intranet, or combinations thereof, and may utilize avariety of networking protocols now available or later developedincluding, but not limited to TCP/IP based networking protocols.

The computer-readable medium 1122 may be a single medium, or thecomputer-readable medium 1122 may be a single medium or multiple media,such as a centralized or distributed database, and/or associated cachesand servers that store one or more sets of instructions. The term“computer-readable medium” may also include any medium that may becapable of storing, encoding or carrying a set of instructions forexecution by a processor or that may cause a computer system to performany one or more of the methods or operations disclosed herein.

The computer-readable medium 1122 may include a solid-state memory suchas a memory card or other package that houses one or more non-volatileread-only memories. The computer-readable medium 1122 also may be arandom access memory or other volatile re-writable memory. Additionally,the computer-readable medium 1122 may include a magneto-optical oroptical medium, such as a disk or tapes or other storage device tocapture carrier wave signals such as a signal communicated over atransmission medium. A digital file attachment to an e-mail or otherself-contained information archive or set of archives may be considereda distribution medium that may be a tangible storage medium.Accordingly, the disclosure may be considered to include any one or moreof a computer-readable medium or a distribution medium and otherequivalents and successor media, in which data or instructions may bestored.

Alternatively or in addition, dedicated hardware implementations, suchas application specific integrated circuits, programmable logic arraysand other hardware devices, may be constructed to implement one or moreof the methods described herein. Applications that may include theapparatus and systems of various embodiments may broadly include avariety of electronic and computer systems. One or more embodimentsdescribed herein may implement functions using two or more specificinterconnected hardware modules or devices with related control and datasignals that may be communicated between and through the modules, or asportions of an application-specific integrated circuit. Accordingly, thepresent system may encompass software, firmware, and hardwareimplementations.

The methods described herein may be implemented by software programsexecutable by a computer system. Further, implementations may includedistributed processing, component/object distributed processing, andparallel processing. Alternatively or in addition, virtual computersystem processing maybe constructed to implement one or more of themethods or functionality as described herein.

Although components and functions are described that may be implementedin particular embodiments with reference to particular standards andprotocols, the components and functions are not limited to suchstandards and protocols. For example, standards for Internet and otherpacket switched network transmission (e.g., TCP/IP, UDP/IP, HTML, HTTP)represent examples of the state of the art. Such standards areperiodically superseded by faster or more efficient equivalents havingessentially the same functions. Accordingly, replacement standards andprotocols having the same or similar functions as those disclosed hereinare considered equivalents thereof.

The illustrations described herein are intended to provide a generalunderstanding of the structure of various embodiments. The illustrationsare not intended to serve as a complete description of all of theelements and features of apparatus, processors, and systems that utilizethe structures or methods described herein. Many other embodiments maybe apparent to those of skill in the art upon reviewing the disclosure.Other embodiments may be utilized and derived from the disclosure, suchthat structural and logical substitutions and changes may be madewithout departing from the scope of the disclosure. Additionally, theillustrations are merely representational and may not be drawn to scale.Certain proportions within the illustrations may be exaggerated, whileother proportions may be minimized. Accordingly, the disclosure and thefigures are to be regarded as illustrative rather than restrictive.

The above disclosed subject matter is to be considered illustrative, andnot restrictive, and the appended claims are intended to cover all suchmodifications, enhancements, and other embodiments, which fall withinthe true spirit and scope of the description. Thus, to the maximumextent allowed by law, the scope is to be determined by the broadestpermissible interpretation of the following claims and theirequivalents, and shall not be restricted or limited by the foregoingdetailed description.

1. A computer-implemented method for valuating responses in acollaborative environment, the method comprising: receiving, by aprocessor, a plurality of responses from a plurality of users based onan item provided by a content provider; receiving, by the processor, aplurality of ratings for each response from the plurality of users;determining, by the processor, a response quality score for eachresponse if a number of the plurality of users who viewed each responsesatisfies a view threshold, wherein the response quality score for eachresponse is based on the plurality of ratings received for each responseand the number of the plurality of users who viewed each response;transforming, by the processor, each response and the response qualityscore of each response into a graphical representation; and providing,by the processor, the graphical representation to a device of thecontent provider.
 2. The method of claim 1 wherein the response qualityscore of each response is determined by dividing a net rating of eachresponse by the number of users who viewed each response.
 3. The methodof claim 2 wherein each rating of the plurality of ratings comprises ofa negative rating or a positive rating and the net rating of eachresponse is determined by subtracting the number of negative ratingsreceived for each response from the number of positive ratings receivedfor each response.
 4. The method of claim 1 wherein the view thresholdis satisfied for each response if the number of the plurality of userswho viewed each response is greater than the number of the plurality ofusers who viewed an amount of the other responses in the plurality ofresponses.
 5. The method of claim 4 wherein the view threshold issatisfied for each response of the plurality of responses if the numberof the plurality of users who viewed each response is greater than thenumber of the plurality of users who viewed at least one-third of theother responses in the plurality of responses.
 6. The method of claim 1wherein transforming, by the processor, each response and the responsequality score of each response into the graphical representation furthercomprises transforming, by the processor, each response and the responsequality score of each response into the graphical representation if thenumber of users who viewed each response satisfies the view threshold.7. The method of claim 1 wherein the graphical representation comprisesa report, wherein each response which does not satisfy the viewthreshold is displayed with an indication that the response was noteligible to be scored.
 8. A computer-implemented method for valuatingresponses of a user in a collaborative environment, the methodcomprising: identifying, by a processor, a collaborative environmentwherein a user provides a plurality of responses based on a plurality ofitems provided by a plurality of content providers, and a plurality ofusers view the plurality of responses and provide a plurality of ratingsfor the plurality of responses; determining, by the processor, aplurality of response quality scores for the plurality of responsesprovided by the user; determining, by the processor, a user responsequality score of the user based on the plurality of response qualityscores for the plurality of responses provided by the user, if aquantity of the responses provided by the user satisfies a contributionthreshold; and storing, by the processor in a data store, the userresponse quality score of the user and a user data describing the userif the quantity of responses provided by the user satisfies acontribution threshold, otherwise storing, by the processor in the datastore, the user data describing the user and an indication that the useris not eligible to be scored.
 9. The method of claim 8 furthercomprising: retrieving, by the processor from the data store, the userresponse quality score of the user and the user data describing the userif the quantity of responses provided by the users satisfies thecontribution threshold, otherwise retrieving, by the processor from thedata store, the user data describing the user and the indication thatthe user is not eligible to be scored; transforming, by the processor,at least the user data and the user response quality score, orindication that the user is not eligible to be scored, into a graphicalrepresentation; and providing, by the processor, the graphicalrepresentation to a device of an administrator.
 10. The method of claim8 wherein determining, by the processor, the user response quality scoreof the user based on the plurality of response quality scores for theplurality of responses provided by the user, if the quantity of theresponses provided by the user satisfies the contribution thresholdfurther comprises determining, by the processor, an average of theplurality of response quality scores for the plurality of responsesprovided by the user if the quantity of the responses provided by theuser satisfies the contribution threshold.
 11. The method of claim 8wherein the response quality score of each response is determined bydividing a net rating of each response by a number of users who viewedeach response.
 12. The method of claim 11 wherein each rating of theplurality of ratings comprises of a negative rating or a positive ratingand the net rating of each response is determined by subtracting thenumber of positive ratings received for each response by the number ofnegative ratings received for each response.
 13. The method of claim 8wherein the contribution threshold is satisfied for the user if thequantity of responses provided by the user is greater than a number ofresponses provided by an amount of other users of the plurality ofusers.
 14. The method of claim 13 wherein the contribution threshold issatisfied for the user if the quantity of responses provided by the useris greater than the number of responses provided by at least one thirdof the users of the plurality of users.
 15. A computer-implementedmethod for valuating users in a collaborative environment, the methodcomprising: determining, by a processor, a contribution rank of a userof the plurality of users in a collaborative environment, wherein thecontribution rank indicates a number of responses provided by the userrelative to other users in the plurality of users; determining, by theprocessor, a rating rank of the user of the plurality of users, whereinthe rating rank indicates a number of ratings provided by the userrelative to the other users in the plurality of users; determining, bythe processor, a user response quality rank of the user of the pluralityof users, wherein the user response quality rank indicates a userresponse quality score of the user relative to the other users;determining, by the processor, a user value of the user based on thecontribution rank, the rating rank and the user response quality rank ofthe user; and providing, by the processor, the user value to a device ofan administrator.
 16. The method of claim 15 wherein determining, by theprocessor, a user value of the user based on the contribution rank, therating rank and the user response quality rank of the user furthercomprises, determining, by the processor, the user value of the user bysumming the contribution rank multiplied by 0.25, the rating rankmultiplied by 0.25, and the user response quality rank multiplied by0.5.
 17. A system for valuating responses in a collaborativeenvironment, the system comprising: a memory to store an item, aplurality of responses and a plurality of ratings for each response; aninterface operatively connected to the memory, the interface operativeto communicate with a plurality of devices of a plurality of users and acontent provider; and a processor operatively connected to the memoryand the interface, the processor operative to receive, via theinterface, the plurality of responses from the plurality of users basedon the item provided by a content provider, receive, via the interface,the plurality of ratings for each response from the plurality of users,determine a response quality score for each response if a number of theplurality of users who viewed each response satisfies a view threshold,wherein the response quality score for each response is based on theplurality of ratings received for each response and the number of theplurality of users who viewed each response, transform each response andthe response quality score of each response into a graphicalrepresentation and provide, via the interface, the graphicalrepresentation to the content provider.
 18. The system of claim 17wherein the processor is further operative to determine the responsequality score of each response by dividing a net rating of each responseby the number of users who viewed each response.
 19. The system of claim18 wherein each rating of the plurality of ratings comprises of anegative rating or a positive rating and the processor is furtheroperative to determine the net rating of each response by subtractingthe number of negative ratings received for each response from thenumber of positive ratings received for each response.
 20. The system ofclaim 17 wherein the view threshold is satisfied for each response ifthe number of the plurality of users who viewed each response is greaterthan the number of the plurality of users who viewed an amount of theother responses in the plurality of responses.
 21. The system of claim20 wherein the view threshold is satisfied for each response of theplurality of responses if the number of the plurality of users whoviewed each response is greater than the number of the plurality ofusers who viewed at least one-third of the other responses in theplurality of responses.
 22. The system of claim 17 wherein the processoris further operative to transform each response and the response qualityscore of each response into the graphical representation if the numberof users who viewed each response satisfies the view threshold.
 23. Thesystem of claim 17 wherein the graphical representation comprises areport, wherein each response that does not satisfy the view thresholdis displayed in the report with an indication that the response was noteligible to be scored.